Skip to content

node-api: widen the measured slice, harden the harness, record the runs to 72.3% - #6

Open
Alan Wisper (wisper-dev) wants to merge 8 commits into
mainfrom
node-api/measured-slice-and-runs
Open

node-api: widen the measured slice, harden the harness, record the runs to 72.3%#6
Alan Wisper (wisper-dev) wants to merge 8 commits into
mainfrom
node-api/measured-slice-and-runs

Conversation

@wisper-dev

@wisper-dev Alan Wisper (wisper-dev) commented Aug 13, 2026

Copy link
Copy Markdown
Member

node-api: widen the measured slice, harden the harness, record the runs

Brings the node-api suite to an honest measurement and records the progression
35.8% → 43.2% → 50.4% → 60.1% → 65.0% → 72.3%. The latest run is 1,542 passes
out of 2,133 counting the denominator without skips.

That run is RED: 18 tests the ratchet expects to pass do not — 7 timeouts,
clustered in http and net, and 11 functional failures. The ratchet is
deliberately left as it was, so those 18 stay visible instead of being absorbed
into the baseline.

Harness

  • Reap the whole process group on timeout. Node's core suite spawns children
    that outlive the test process and hold its stdout open, so killing the entry
    process alone left the output read hanging and the run stalled. The process now
    gets its own group and a timeout reaps the tree. A descendant that opened its
    own session still escapes the group kill, so the pipes are dropped after a
    two-second grace.
  • Dereference the local install when staging the image. --elide usually
    points at the dist/current symlink, and the image build context must carry
    the real files; the staged copy now dereferences instead of building around a
    broken link.
  • The cgroup cage is optional. Probing systemd-run no longer fails the run:
    Bun.spawnSync throws for a missing executable rather than reporting an exit
    code, so inside the harness container the whole suite died before its first
    test. An absent systemd-run now means "run uncaged" and the container's own
    limits apply. A test covers that branch: it strips the PATH of the child it
    spawns rather than mutating process.env, because the search path a process
    uses is the one it started with.
  • Registry: the test expects the configured jtreg case timeout rather than
    the previous default.

Test selection

  • Coverage widened with the missing subsystem groups: child_process, cluster,
    dgram, domain, http, https, net, os, perf_hooks, querystring,
    readline, tty, v8, zlib and more, so the suite measures the public Node
    surface rather than a sparse slice.
  • Memory bound per test: -XX:MaxHeapSize=4g. The suite contains cases whose
    only brake is stream backpressure (test-zlib-brotli-16GB.js decompresses
    16 GB); unbounded, such a case exhausts the host instead of failing.
  • New static expectations file expectations/node-api.toml carries the skips with
    a reason each: the cluster group is out of project scope for now (one glob, so
    the decision stays visible and reversible), and the node:vfs tests cover an
    experimental Node subsystem Elide does not implement.

Ratchet and reports

  • The ratchet holds 857 known failures. It is deliberately not retaken against
    the newest run, so the 18 regressions above stay reported rather than absorbed.
  • Six runs are recorded. The latest carries its full report body — the impact
    map is the working document for the next batch of fixes. The earlier ones
    keep only the raw results, counts and change summary: that is what the trend,
    the run database (ingestAll) and the next run's diff actually read, while the
    rendered bodies are superseded as soon as a newer run lands.
  • BRIEFING-node-api.md is rewritten around the current measurement: the failure
    map by subsystem with the hang subset broken out, what the static expectations
    skip and why, the ranked work items that follow, and run instructions matching
    the current selection.

Checks

bun --cwd harness test — 172/172 pass. bun run typecheck — clean.

Node's core suite spawns children that outlive the test process and hold its
stdout open, so killing the entry process alone left the read hanging. The
process now runs in its own group and a timeout reaps the tree; a descendant
that opened its own session escapes the kill, so the pipes are dropped after a
two-second grace.
`--elide` usually points at the `dist/current` symlink, and the image build
context must carry the real files; the staged copy now dereferences instead of
building around a broken link.
The workload sets its own case timeout; the test asserted the previous default.
Adds the missing subsystem groups (child_process, cluster, dgram, domain, http,
net, os, readline, tty, zlib and more) so the suite measures the public Node
surface rather than a sparse slice.

Each test now runs with a heap bound: the suite contains cases whose only brake
is stream backpressure (test-zlib-brotli-16GB.js decompresses 16 GB), and
unbounded such a case exhausts the host instead of failing.

A static expectations file carries the skips with a reason each: the cluster
group is out of project scope for now, and the node:vfs tests cover an
experimental Node subsystem Elide does not implement.
Records the measured progression on the widened slice: 35.8%, 43.2%, 50.4%,
60.1% and 65.0% (1,387 of 2,133 counting the denominator without skips). The
latest run carries its full report body; the earlier ones keep the raw results,
counts and change summary, which is what the trend, the run database and the
next run's diff read. The ratchet is retaken against the latest run and holds
857 known failures.
Replaces the July handoff doc: the failure map by subsystem with the hang subset
broken out, what the static expectations skip and why, the ranked work items
that follow, and run instructions that match the current selection.
Node's suite contains tests that fork without bound, and the timeout-driven
process-group kill only fires once a runaway has already taken the machine's
memory. Each test now runs in a transient systemd scope with memory and task
ceilings plus a runtime cap that reaps the whole cgroup, descendants that
called setsid included; the harness container gets its own memory and pid
limits with swap disabled. Where cgroups are unavailable — inside the
container, on macOS — the run proceeds as before.
Probing the cgroup cage no longer fails the run: `Bun.spawnSync` throws for a
missing executable rather than reporting an exit code, so inside the harness
container — where `systemd-run` does not exist — the whole suite died before its
first test. An absent `systemd-run` now means "run uncaged", as intended, and the
container's own limits apply. A test covers that branch: it strips the PATH of the
child it spawns rather than mutating `process.env`, because the search path a
process uses is the one it started with.

The run itself, on Elide 1.4.4+405f52319: 1,542 of 2,133 excluding skips — 72.3%,
up from 65.0%, with 577 new passes and 194 fixed. It is RED: 18 tests the ratchet
expects to pass do not, 7 of them timeouts clustered in http and net, and 11
functional, led by `test-stream-pipeline-duplex` and
`test-stream-pipe-flow-after-unpipe`. The ratchet is deliberately left as it was,
so those 18 stay visible rather than being absorbed into the baseline.

The briefing is rewritten around the new numbers: the failure map (591 files, http
now the largest block at 79 with 26 hangs), the largest root-cause signatures, and
a work order that puts the regressions first.
@wisper-dev Alan Wisper (wisper-dev) changed the title node-api: widen the measured slice, harden the harness, record the runs to 65% node-api: widen the measured slice, harden the harness, record the runs to 72.3% Aug 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant